Variational inference has recently emerged as a popular alternative to the classical Markov chain Monte Carlo (MCMC) in large-scale Bayesian inference. The core idea is to trade statistical accuracy for computational efficiency. In this work, we study these statistical and computational trade-offs in variational inference via a case study in inferential model selection. Focusing on Gaussian inferential models (or variational approximating families) with diagonal plus low-rank precision matrices, we initiate a theoretical study of the trade-offs in two aspects, Bayesian posterior inference error and frequentist uncertainty quantification error. From the Bayesian posterior inference perspective, we characterize the error of the variational posterior relative to the exact posterior. We prove that, given a fixed computation budget, a lower-rank inferential model produces variational posteriors with a higher statistical approximation error, but a lower computational error; it reduces variance in stochastic optimization and, in turn, accelerates convergence. From the frequentist uncertainty quantification perspective, we consider the precision matrix of the variational posterior as an uncertainty estimate, which involves an additional statistical error originating from the sampling uncertainty of the data. As a consequence, for small datasets, the inferential model need not be full-rank to achieve optimal estimation error (even with unlimited computation budget).

I hope the previous part (Part I) was useful for you or made any impact in your current life because I know how much effort requires start anything new and keep into, but the main reason of this kind of stories are remarke the importance about data science and machine learning in IT progress world where data and datasets are the main dish in menu. The world is changing and the focus in AI too. In this chat, Andrew Ng (Deep Learning specialist, Founder Landing AI and Deeplearning.AI) share the skills he see as fundamental to the next generation of machine learning practitioners (link chat video). He talk about the "old vision or approach" in model-centric: Passionately work on new algorithms, mathematical formulas, meta-architectures, convolutional layer stacking with normalization and all the study of inferential models and their components. But today most architectures are tested with optimal results, it is known that the application of a convolutional architecture is key to later achieve classification, object detection or segmentation, the power of LSTM (long short term memory) is known to language processing applications such as time series (real-time vehicle self-driving). So continuing on the path of algorithm-oriented improvements is no relevant.

A fundamental problem in statistics and machine learning is that of using observed data to predict future observations. This is particularly challenging for model-based approaches because often the goal is to carry out this prediction with no or minimal model assumptions. For example, the inferential model (IM) approach is attractive because it has certain validity guarantees, but requires specification of a parametric model. Here we show that a new perspective on a recently developed generalized IM approach can be applied to construct an IM for prediction that satisfies the desirable validity guarantees without specification of a model. One important special case of this approach corresponds to the powerful conformal prediction framework and, consequently, the desirable properties of conformal prediction follow immediately from the general IM validity theory. Several numerical examples are presented to illustrate the theory and highlight the method's performance and flexibility.